Media recognition and synchronisation to a motion signal

ABSTRACT

The present document describes a device and method for synchronizing a motion signal corresponding to a media content with a media signal for the media content, the motion signal being for controlling a motion feedback system. The method comprises: receiving a portion of the media signal; obtaining a fingerprint corresponding to the received portion of the media signal; from reference fingerprints associated with time positions of at least one reference media content, identifying a reference time position of the media content corresponding to the obtained fingerprint; obtaining the motion signal associated with the identified reference time position of the media content; and outputting the motion signal synchronized with the media signal using the identified reference time position of the media content for controlling the motion feedback system.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority of U.S. provisional application61/366,301 filed Jul. 21, 2010 and entitled “MEDIA RECOGNITION ANDSYNCHRONISATION TO A MOTION SIGNAL”, the content of which isincorporated herein by reference.

TECHNICAL FIELD

This description relates to the field of media recognition andsynchronization. More particularly, this description relates to methodsand devices for playing video and audio stream media in synchronizationwith a motion signal.

BACKGROUND

In controlling motion feedback systems for use in public, commercial orhome environments, the challenge is in the synchronization of a motionsignal for gaming, feature length movies, amusement park rides, virtualreality, etc. with audio or video signals during playback, such that themotion signalis delivered to motion feedback systems in time for theplatform movement to match the reproduced audio-visual signals. Therequired tolerance in usually in the order of milliseconds.

In some applications, the motion signal is not transmitted synchronouslywith audio or video signals; the playback device must retrieve themotion signal separately, and synchronize them for playback with theaudio or video signals. If the identity of the audio or video signalsand position in time within the audio or video signals can be readilyand accurately identified through metadata in the audio or videosignals, the playback device may use this information to retrieve andsynchronize the appropriate motion signal. If such metadata is notavailable, then another solution is required.

There is therefore a need for an improved synchronization method anddevice which addresses the above-noted challenges.

SUMMARY

The present disclosure therefore provides a motion signalsynchronization method and device that overcomes or mitigates one ormore disadvantages known to be associated with the prior art, or atleast provides a useful alternative.

The proposed approach is based on a technique commonly referred to asmedia fingerprinting. This technique enables media recognition by way ofderiving unique identifiers, which are typically referred to as mediafingerprints, from particular media content. The correspondence betweenparticular media fingerprints and particular media content is then usedfor identification purposes. For example, the recognition of audio andvideo content exploits acoustic and visual features to identifyparticular corresponding acoustic and video media respectively. Knownmedia fingerprinting techniques are described in the following patentreferences: WO2002065782 (Philips), US20060041753 (Philips), andUS20060075237 (Philips).

In light of the above media fingerprinting technique, the motion signalsynchronization herein proposed employs media fingerprinting torecognize a given portion of an audio or video signal, to which is thenassociated a given motion signal for use during playback.

It was found that applying media fingerprinting techniques to identify areference time position of the media content (e.g., the time position ina movie which is presently being read or played) and then obtain themotion signal which corresponds to this reference time position providesa satisfactory result since the media content and the motion signal canthen be outputted synchronously. This combination provides a simpler,more precise, more robust, more stable, less expensive, morefault-tolerant solution that those described in the prior art forsynchronizing a motion signal with a media signal. In contrast to priorart method, this method is completely independent of the method used forencoding, compressing or packaging the audio signal. Because this methodrelies on the acoustic characteristics of the audio signal, it can bemade resilient to changes in the pitch, speed or amplitude thereof.

Motion feedback systems are meant to include motion platforms, motionactuators, motion simulator, vibrotactile transducers, and inertialshakers.

Motion platforms are meant to include any platform or seatingarrangement to which motion is induced by actuators and on which aperson is installed. Motion platform applications include, but are notlimited to, gaming, feature length movies, amusement park rides, virtualreality, etc. An example of a motion platform include seat or chair forone or more persons on which are mounted one or more actuators whichinterface with the ground. Another example would be a platform forreceiving a seat, chair or other device accommodating a user, and onwhich are mounted one or more actuators which interface with the ground.

Motion codes, a motion code signal or a motion code stream containmotion effects, based on the content meant to be reproduced to theviewer. In most cases, the motion signals are prepared by motiondesigners. In other cases, the motion signals or portion thereof aregenerated automatically.

A media signal is to be understood as comprising an audio signal and/ora video signal.

An audio signal or an audio stream is to be understood as encompassingsound information meant to be reproduced to the during the presentation,such as the sound track of games, feature length movies, amusement parkrides, virtual reality environments, etc. An example of an audio signalor an audio signal is encoded Pulse Code Modulation (PCM) samples.

A video signal or a video stream is to be understood as encompassingvisual information meant to be reproduced to the user, such as the mainpicture of games, feature length movies, amusement park rides, virtualreality environments, etc. An example of a video signal or a videosignal is an MPEG-4 Part 10 video elementary stream.

Media content is meant to include games, feature length movies,amusement park rides, virtual reality environments, advertisements, etc.

Feature length movies are those intended to be shown in movie theatersor home theaters. They last generally more than 30 minutes and haveseparate video and audio tracks to which a motion signal, a stream ofmotion codes or a motion code track can be associated.

According to an embodiment, there is provided a method for synchronizinga motion signal corresponding to a media content with a media signal forthe media content, the motion signal for controlling a motion feedbacksystem, the method comprising: receiving a portion of the media signal;obtaining a fingerprint corresponding to the received portion of themedia signal; from reference fingerprints associated with time positionsof at least one reference media content, identifying a reference timeposition of the media content corresponding to the obtained fingerprint;obtaining the motion signal associated with the identified referencetime position of the media content; and outputting the motion signalsynchronized with the media signal using the identified reference timeposition of the media content for controlling the motion feedbacksystem.

According to another embodiment, there is provided a method fordetermining if a motion signal corresponding to a media content issynchronized to a media signal for the media content, the motion signalfor controlling a motion feedback system, the method comprising:receiving a portion of the media signal; obtaining a fingerprint of thereceived portion of the media signal, namely an obtained fingerprint;retrieving a fingerprint of the portion of the media signal at areference time position of the media content, namely a referencefingerprint; comparing the obtained fingerprint and the referencefingerprint; based on the comparing, determining if the motion signal issynchronized to the media signal; and according to the determining,outputting the motion signal synchronized with the media signal forcontrolling the motion feedback system.

According to another embodiment, there is provided a device forsynchronizing a motion signal corresponding to a media content with amedia signal for the media content, the device having access to astorage medium which comprises reference fingerprints associated withtime positions of at least one reference media content, the motionsignal for controlling a motion feedback system, the device comprising:an input unit for receiving a portion of the media signal; a processingdevice in communication with the input unit and the storage medium; anda memory accessible by the processing device, the memory storinginstructions for implementing the processing device to: obtain afingerprint corresponding to the received portion of the media signal;from reference fingerprints associated with time positions of at leastone reference media content, identify a reference time position of themedia content corresponding to the obtained fingerprint; obtain themotion signal associated with the identified reference time position ofthe media content; and output the motion signal synchronized with themedia signal using the identified reference time position of the mediacontent for controlling the motion feedback system.

According to another embodiment, there is provided a non-transitorycomputer readable memory having recorded thereon statements andinstructions for execution by a computer to carry out a method forsynchronizing a motion signal corresponding to a media content with amedia signal for the media content, the motion signal for controlling amotion feedback system, the method comprising: receiving a portion ofthe media signal; obtaining a fingerprint corresponding to the receivedportion of the media signal; from reference fingerprints associated withtime positions of at least one reference media content, identifying areference time position of the media content corresponding to theobtained fingerprint; obtaining the motion signal associated with theidentified reference time position of the media content; and outputtingthe motion signal synchronized with the media signal using theidentified reference time position of the media content for controllingthe motion feedback system.

BRIEF DESCRIPTION OF THE DRAWINGS

Further features and advantages of the present disclosure will becomeapparent from the following detailed description, taken in combinationwith the appended drawings, in which:

FIG. 1 is a flowchart of a method for synchronizing a motion signal toan audio signal, in accordance with an embodiment;

FIG. 2 is a flowchart which illustrates an embodiment of a method fordetermining if a motion signal corresponding to a media content issynchronized to a media signal for the media content; and

FIG. 3 is a schematic illustration of a device for synchronizing amotion signal to an audio signal, in accordance with an embodiment.

It will be noted that throughout the appended drawings, like featuresare identified by like reference numerals.

DETAILED DESCRIPTION

Referring now to the drawings, and more particularly to FIG. 1, there isillustrated an embodiment of a method 20 for synchronizing a motionsignal corresponding to a media content, for example from a featurelength movie or amusement park rides, with a media signal for the mediacontent. The motion signal is for controlling a motion feedback system.According to an embodiment, the media signal is an audio signal and/or avideo signal.

In step 22, a portion of the media signal is received. In step 24, afingerprint corresponding to the received portion of the media signal isobtained.

According to an embodiment, the step of obtaining a fingerprintcorresponding to the received portion of the media signal (step 24)comprises computing the fingerprint of the received portion of the mediasignal. In the computing the fingerprint step, the fingerprint of theportion of media signal may be generated as a code that uniquelycorresponds to a waveform of the media signal. Upon generating such aunique code (i.e., a coded identifier), the corresponding waveform ofthe media signal from which the fingerprint was generated is thereafteridentifiable by referencing to its fingerprint as described herein.

In step 26, from reference fingerprints associated with time positionsof at least one reference media content, a reference time position ofthe media content corresponding to the obtained fingerprint isidentified.

More specifically, in one embodiment of step 26, a database (aka astorage medium) of coded identifiers for the fingerprints for the mediasignal at the multiple playback time positions is consulted. Thefingerprint of the portion of the media signal is matched to a referencefingerprint associated to the reference time position within the mediacontent.

According to an embodiment, the identifying a reference time position ofthe media content (step 26) takes place on a remote networked server(not shown).

According to an embodiment, method 20 further comprises receiving theidentity of one reference media content of the at least one referencemedia content to which the media signal corresponds prior to theidentifying a reference time position of the media content (step 26).The step of identifying the reference time position of the media contentis performed using reference fingerprints associated with time positionsof the identified reference media content.

Method 20 may further comprises receiving at least one referencefingerprint associated with the outputted motion signal and stopping theoutput of the motion signal when the obtained fingerprint does not matchany of the at least one reference fingerprint.

In another embodiment of step 24, the obtaining the fingerprintcomprises retrieving the coded identifier for the fingerprint of theportion of the media signal at the reference time position directly fromthe motion signal.

In step 28, the motion signal associated with the identified referencetime position of the media content is obtained. According to anembodiment, the motion signal associated with the identified referencetime position of the media content is obtained from a remote networkedserver (not shown).

According to an embodiment of step 28, the identifying the currentplayback time position comprises continuously performing the obtaining afingerprint, the identifying a reference time position, and theobtaining the motion signal steps to keep outputting the motion signalsynchronized with the media signal, thereby ensuring continuoussynchronization of the motion signal.

According to another embodiment of step 28, the identifying the currentplayback time position comprises performing the obtaining a fingerprint,the identifying a reference time position, and the obtaining the motionsignal at a given frequency to resynchronize the outputting the motionsignal with the media signal.

The reference time position refers to the location of the media signalwithin the at least one media content which is identified ascorresponding to media content which is currently being played(corresponding to the obtained fingerprint) on either one of thespeakers (for audio), the screen or display (for video) and the motionfeedback system (for motion).

Step 30 involves outputting the motion signal, using the identifiedreference time position of the media content, synchronized with themedia signal. The motion signal is eventually sent to a motion feedbacksystem having actuators for moving the motion feedback system accordingto the motion signal.

According to an embodiment, method 20 further comprises receiving amotion signal corresponding to the media content prior to theidentifying a reference time position of the media content (step 26),wherein the reference fingerprints are retrieved directly from thereceived motion signal.

According to an embodiment, the synchronized motion signal is stored ina storage device for eventually being sent to the motion feedbacksystem.

The method 20 is performed, in one example, in real-time, as the mediasignal is being played.

Still according to another embodiment, method 20 further comprisesreceiving at least one reference fingerprint associated with theoutputted motion signal and stopping the output of the motion signalwhen the obtained fingerprint does not match any of the at least onereference fingerprint.

FIG. 2 illustrates an embodiment of a method 100 for determining if amotion signal corresponding to a media content is synchronized to amedia signal for the media content.

In step 102, a portion of the media signal is received. In step 104, afingerprint of the media signal for the current playback time positionis obtained (i.e., an obtained fingerprint).

In step 106, a fingerprint of the portion of the media signal at areference time position of the media content is retrieved based on thecurrent position within the motion signal, namely a referencefingerprint.

In step 108, the reference fingerprint of step 106 is compared with theobtained fingerprint of step 104. In step 110, based on the comparing,it is determined if the motion signal is synchronized to the mediasignal If the two fingerprints differ, the motion signal and mediasignal are not synchronized. This can occur, for instance, if theposition within the motion signal changes abruptly, such as during afast-forward or chapter-skip operation. The system may automaticallychoose as a result to apply method 20 to recover synchronization.

In another embodiment of step 104, the obtaining the fingerprintcomprises retrieving the coded identifier for the fingerprint of theportion of the media signal at the reference time position directly fromthe motion signal.

In step 112, the motion signal synchronized with the media signal usingthe identified reference time position of the media content forcontrolling the motion feedback system is outputted according to theresult of the determining if the motion signal is synchronized with themedia signal (step 110).

Now referring to FIG. 3, there is shown an embodiment of a device 50 forsynchronizing a motion signal for a feature length movie with a mediasignal.

The device 50 has an input unit 52; a processing unit 54; a memory 56;one or more databases (here databases 58 and 60); and an output unit 62in communication with a motion feedback system 64 having actuators 1 toN, 66 to 68.

The input unit 52 is for receiving a portion of a media signal.

The databases 58 and 60 are used respectively for storing motion signalsand reference media signal fingerprints in association with particularknown media content (e.g., a known feature length movie). Such referencefingerprints are identifiers already derived for the known mediacontent, and which are known to be uniquely related to the known mediacontent.

The processing unit 54 is in communication with the input unit 52 andthe databases 58 and 60, as well as with the memory 56. The memory 56 isthus accessible by the processing unit 54.

In one embodiment, the memory 56 is used to store instructions forimplementing the processing unit 54 to perform a series of steps on themedia signal which involve access and retrieval of data from thedatabases 58 and 60, as well as processing steps allowing thesynchronisation of a motion signal with the media signal. The processingunit 54 is also in communication with the output unit 62 such that themotion signal, once retrieved and synchronized, are sent out either forstorage on a database (not shown), or directly to a motion feedbacksystem 64 during playback of the media signal.

Once the portion of the media signal is received at the processing unit54, via its transmission from the input unit 52, the processing unit 54obtains a fingerprint corresponding to the received portion of the mediasignal.

The processing unit 54 then proceeds to identify, from referencefingerprints associated with time positions of at least one referencemedia content, a reference time position of the media contentcorresponding to the obtained fingerprint. For example, to perform suchcomparison, the processing unit 54 accesses the database 60 of referencefingerprints; parses them via optional retrieval of data to an optionalinternal memory (not shown) until a match is detected between areference media signal fingerprint and the fingerprint of the mediasignal. When no match is detected, an error signal is sent out to a userinterface (not shown).

Then the processing device obtains the motion signal associated with theidentified reference time position of the media content.

Then, based on the identified reference time position of the mediacontent, the processing device is able to synchronize and output themotion signal with the media signal. For example, to do so, theprocessing unit 54 retrieves and sends the motion signal from thedatabase 58 starting at the reference time position that corresponds tothe reference time of the media signal received at input unit 52. Themotion signal which is sent at output unit 62 is therefore synchronizedwith the playback time position of the media signal.

According to another embodiment, there is provided a non-transitorycomputer readable memory (not shown) having recorded thereon statementsand instructions for execution by a computer to carry out a method forsynchronizing a motion signal corresponding to a media content with amedia signal for the media content. The motion signal is for controllinga motion feedback system. The method comprises method 20 shown in FIG.1.

Finally, according to another embodiment, there is provided anon-transitory computer readable memory (not shown) having recordedthereon statements and instructions for execution by a computer to carryout a method for determining if a motion signal corresponding to a mediacontent is synchronized to a media signal for the media content. Themotion signal is for controlling a motion feedback system. The methodcomprises method 100 shown in FIG. 2.

While embodiments have been described above and illustrated in theaccompanying drawings, it will be evident to those skilled in the artthat modifications may be made therein without departing from the scopeof this disclosure. Such modifications are considered as possiblevariants comprised in the scope of the disclosure.

1. A method for synchronizing a motion signal corresponding to a mediacontent with a media signal for the media content, the motion signal forcontrolling a motion feedback system, the method comprising: receiving aportion of the media signal; obtaining a fingerprint corresponding tothe received portion of the media signal; from reference fingerprintsassociated with time positions of at least one reference media content,identifying a reference time position of the media content correspondingto the obtained fingerprint; obtaining the motion signal associated withthe identified reference time position of the media content; andoutputting the motion signal synchronized with the media signal usingthe identified reference time position of the media content forcontrolling the motion feedback system.
 2. The method of claim 1,further comprising receiving the identity of one reference media contentof the at least one reference media content to which the media signalcorresponds prior to the identifying a reference time position of themedia content, and wherein the step of identifying a reference timeposition of the media content is performed using reference fingerprintsassociated with time positions of the identified reference mediacontent.
 3. The method of claim 2, further comprising receiving at leastone reference fingerprint associated with the outputted motion signaland stopping the output of the motion signal when the obtainedfingerprint does not match any of the at least one referencefingerprint.
 4. The method of claim 1, further comprising receiving atleast one reference fingerprint associated with the outputted motionsignal and stopping the outputting the motion signal when the obtainedfingerprint does not match any of the at least one referencefingerprint.
 5. The method of claim 1, wherein the media signalcomprises at least one of an audio signal and a video signal.
 6. Themethod of claim 5, wherein the media content comprises a feature lengthmovie and the audio signal and the video signal comprise the audiosignal and the video signal for a feature length movie.
 7. The method ofclaim 1, wherein the obtaining a fingerprint corresponding to thereceived portion of the media signal comprises computing the fingerprintof the received portion of the media signal.
 8. The method of claim 7,wherein the computing the fingerprint comprises applying a mediafingerprinting technique to generate the fingerprint as a codedidentifier uniquely representative of the media content at the referencetime position.
 9. The method of claim 8, wherein the identifying areference time position of the media content comprises consulting adatabase comprising coded identifiers for the reference fingerprints towhich are associated reference time positions and obtaining thereference time position that corresponds to the coded identifieruniquely representative of the media content at the reference timeposition.
 10. The method of claim 1, wherein the identifying thereference time position comprises continuously performing the obtaininga fingerprint, the identifying a reference time position, and theobtaining the motion signal steps to keep outputting the stream ofmotion signal synchronized with the media signal, thereby ensuringcontinuous synchronization of the motion signal.
 11. The method of claim1, wherein the identifying the reference time position comprisesperforming the obtaining a fingerprint, the identifying a reference timeposition, and the obtaining the motion signal at a given frequency toresynchronize the outputting the motion signal with the media signal.12. The method of claim 1, further comprising receiving a motion signalcorresponding to the media content prior to the identifying a referencetime position of the media content, wherein the reference fingerprintsare retrieved directly from the received motion signal.
 13. The methodof claim 1, wherein the identifying a reference time position of themedia content takes place on a remote networked server.
 14. The methodof claim 1, wherein the motion signal associated with the identifiedreference time position of the media content is obtained from a remotenetworked server.
 15. A method for determining if a motion signalcorresponding to a media content is synchronized to a media signal forthe media content, the motion signal for controlling a motion feedbacksystem, the method comprising: receiving a portion of the media signal;obtaining a fingerprint of the received portion of the media signal,namely an obtained fingerprint; retrieving a fingerprint of the portionof the media signal at a reference time position of the media content,namely a reference fingerprint; comparing the obtained fingerprint andthe reference fingerprint; based on the comparing, determining if themotion signal is synchronized to the media signal; and according to thedetermining, outputting the motion signal synchronized with the mediasignal for controlling the motion feedback system.
 16. The method ofclaim 15, wherein the reference fingerprint is retrieved from a databasecomprising reference fingerprints associated with time positions for themedia content.
 17. The method of claim 16, wherein the referencefingerprint is retrieved directly from the motion signal.
 18. A devicefor synchronizing a motion signal corresponding to a media content witha media signal for the media content, the device having access to astorage medium which comprises reference fingerprints associated withtime positions of at least one reference media content, the motionsignal for controlling a motion feedback system, the device comprising:an input unit for receiving a portion of the media signal; a processingdevice in communication with the input unit and the storage medium; anda memory accessible by the processing device, the memory storinginstructions for implementing the processing device to: obtain afingerprint corresponding to the received portion of the media signal;from reference fingerprints associated with time positions of at leastone reference media content, identify a reference time position of themedia content corresponding to the obtained fingerprint; obtain themotion signal associated with the identified reference time position ofthe media content; and output the motion signal synchronized with themedia signal using the identified reference time position of the mediacontent for controlling the motion feedback system.
 19. A non-transitorycomputer readable memory having recorded thereon statements andinstructions for execution by a computer to carry out a method forsynchronizing a motion signal corresponding to a media content with amedia signal for the media content, the motion signal for controlling amotion feedback system, the method comprising: receiving a portion ofthe media signal; obtaining a fingerprint corresponding to the receivedportion of the media signal; from reference fingerprints associated withtime positions of at least one reference media content, identifying areference time position of the media content corresponding to theobtained fingerprint; obtaining the motion signal associated with theidentified reference time position of the media content; and outputtingthe motion signal synchronized with the media signal using theidentified reference time position of the media content for controllingthe motion feedback system.